The present invention relates to a multimedia documents integrating and displaying system for handling information in a wide variety of formats including images, video, audio and three-dimensional graphics in a unified fashion and using them in a wide variety of combinations.
Recently, the amount of information available to an Internet user through the World Wide Web (hereafter, referred to as xe2x80x9cWebxe2x80x9d) has been increasing exponentially. Concomitantly, the variety of formats for information, such as text, images, video, audio, three-dimensional graphics and the like, has been rapidly increasing.
However, methods for handling all information formats in a unified environment are only at an early stage of development. Especially, there is a great need for a method of handling various information formats in a unified three-dimensional space.
An object of the present invention, created in consideration of the above circumstances, is to provide a multimedia documents integrating and displaying system capable of handling various formats for information including images, video, audio and three-dimensional graphics. The aforesaid system uses an extended function of Extended Markup Language (XML) in a language specification of a unified description, and effectively combines data groups scattered across the Web to thereby allow an Web content creator or an Web user to retrieve and use required information from various and large amount of information in an optimal form.
More specifically, an object of the present invention is to provide a multimedia documents integrating and displaying system, which allows its user to specify sound, three-dimensional images, two-dimensional images and the like, and build an environment which operates at a comfortable speed without requiring single-purpose viewers and players such as plug-ins and the like.
To achieve the aforesaid objectives, according to a primary aspect of the present invention, there is provided a multimedia documents integrating and displaying system, comprising: a framework means for operating in close communication with a parser, interpreting various drawing tags in a document described in a markup language, generating a framework for synchronizing and operating various drawing primitives, and generating a two-dimensional drawing command and a three-dimensional drawing command based on the aforesaid framework; a two-dimensional rendering means for receiving the aforesaid two-dimensional drawing command from the aforesaid framework means, and creating a two-dimensional canvas capable of executing the two-dimensional drawing command on a Web browser; and a three-dimensional rendering means for receiving the aforesaid three-dimensional drawing command from the aforesaid framework means, and creating a three-dimensional canvas capable of executing the three-dimensional drawing command on the Web browser. Here, the aforesaid markup language is preferably XML. Also it is desirable that the aforesaid two-dimensional rendering means and the aforesaid three-dimensional rendering means generates graphic display data for describing a two-dimensional image or a three-dimensional image using a command set in Java AWT; and passing the graphic display data to the Web browser.
According to such a structure, it is possible to build an environment in which all functions are written in a descriptive language such as Java and the like, and easily specify sound, three-dimensional images and two-dimensional images in a framework of a markup language such as XML and the like, thereby eliminating the necessity for single-purpose viewers and players on the Web browser.
According to one embodiment of the present invention, the aforesaid framework means comprises: an execution request generation means, which is activated by the parser, for receiving call-back data corresponding to a predetermined DTD from the parser, and generating execution request description data including a description type to which a parameter is added; and a manager means for generating a two-dimensional drawing command and a three-dimensional drawing command based on the aforesaid execution request description data.
In the above case, it is preferable that the aforesaid execution request generation means comprises: an SVG means for receiving from the aforesaid parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of SVG, and generating execution request description data which indicates a two-dimensional drawing, sound, a moving image, an image, text or a hyperlink.
Also it is preferable that the aforesaid execution request generation means comprises: a CMW means for receiving from the aforesaid parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD of a CMW (Cross-Media World) newly set to suit the present invention, and generating execution request description data which indicates a material of a three-dimensional graphic form, a texture, a camera, a light, a font, an interpolator, a timer, a sensor, image mapping, an XVL file or a VRML file.
Furthermore, it is preferable that the aforesaid execution request generation means comprises: a TVX means for receiving from the aforesaid parser, callback data which indicates contents of an element, an attribute name and a value based on a DTD in TVX (TVML with XML) newly set to suit the present invention, and generating execution request description data which indicates sound, a moving image, an image, a material of a three-dimensional graphic form, a texture, a camera, a light, an interpolator, a timer, a sensor or a VRML file.
According to the above structure, all functions can be described with an XML document, and a plurality of media in different formats can be easily combined.
According to the one embodiment, the aforesaid manager means comprises the means of: structuring the aforesaid execution request description data to thereby create a tree structure for an entire document described in a markup language; and generating a two-dimensional drawing command or a three-dimensional drawing command according to the execution request description data in the aforesaid tree structure to thereby express the document.
In this case, it is preferable that the aforesaid manager means further comprises the means of: mixing a two-dimensional drawing description and a three-dimensional drawing description according to a location of the execution request description data in the aforesaid tree structure; and generating a two-dimensional embedding command for embedding a two-dimensional canvas provided by the aforesaid two-dimensional rendering means in a three-dimensional canvas provided by the aforesaid three-dimensional rendering means.
According to such a structure, a two-dimensional description and a three-dimensional description can be easily combined, and therefore, a two-dimensional image can be embedded in the three-dimensional canvas. Concomitantly, the two-dimensional image embedded in the three-dimensional canvas can retain two-dimensional attributes therein.
According to the one embodiment, the aforesaid manager means further comprises: a means for converting an operation event received through the Web browser to a two-dimensional drawing command or a three-dimensional drawing command. According to such a structure, it is possible to receive an operation event from a user and generate a two-dimensional drawing command or a three-dimensional drawing command according to the operation event.
According to the one embodiment, the aforesaid framework means, the two-dimensional rendering means and the three-dimensional rendering means comprise the means of: being activated by the Web browser; and being executed after being read as objects. According to such a structure, all functions can be downloaded on demand from an Web server for execution.
According to the one embodiment, the aforesaid two-dimensional rendering means comprises: a means for receiving a two-dimensional drawing command from the aforesaid manager means, and generating a drawing method call according to a content of the two-dimensional drawing command; a two-dimensional drawing means for receiving the aforesaid drawing method call, and converting the drawing method call to media operation data; and a two-dimensional canvas means for receiving the media operation data from the two-dimensional drawing means, generating a two-dimensional canvas on the Web browser, and executing a graphic display according to the media operation data.
According to such a structure, it is possible to generate media operation data according to a two-dimensional processing type (static image, moving image, text or the like) and execute the media operation data on a two-dimensional canvas on the Web browser.
According to the one embodiment, the aforesaid two-dimensional canvas means comprises: a two-dimensional object managing means for generating a rendering request after receiving the media operation data and transferring a two-dimensional file to be used by the aforesaid two-dimensional drawing means, and requesting image data generation for an entire two-dimensional canvas based on the media operation data and the two-dimensional file; and a two-dimensional object display means for receiving the rendering request from the aforesaid two-dimensional object managing means, and setting a two-dimensional display attribute in a two-dimensional canvas according to the rendering request.
In this case, it is preferable that the aforesaid two-dimensional canvas means comprises: a two-dimensional media operation means for receiving an operation event for a two-dimensional operation from the Web browser, and passing a rendering request to the aforesaid two-dimensional object display means according to the operation event. Further in this case, it is desirable that the aforesaid two-dimensional object display means comprises the means of: passing a partial rendering request to the aforesaid two-dimensional drawing means according to the rendering request received from the aforesaid two-dimensional media operation means; and receiving the media operation data from the two-dimensional drawing means.
According to the one embodiment, the aforesaid two-dimensional canvas means comprises: a means for calculating transformation data for a two-dimensional canvas if the two-dimensional canvas is pasted to a three-dimensional canvas, and setting a two-dimensional display attribute on a transformed two-dimensional canvas according to the transformation data.
According to such a structure, a display attribute of the two-dimensional canvas can be set on a three-dimensional canvas.
According to the one embodiment, the aforesaid three-dimensional rendering means comprises: a three-dimensional scene graph generation means for reading three-dimensional data based on three-dimensional drawing commands received from the aforesaid manager means, and generating three-dimensional scene graphs for three-dimensional displays according to the three-dimensional drawing commands; a three-dimensional scene drawing means for retaining a scene graph tree, which is an accumulation of the above three-dimensional scene graphs, and generating scene drawing data by traversing the scene graph tree; and a three-dimensional canvas means for generating a three-dimensional canvas on a predetermined drawing area on the Web browser, and concomitantly generating graphic display data to be displayed on the aforesaid three-dimensional canvas based on three-dimensional scene drawing data from the aforesaid three-dimensional scene drawing means.
According to such a structure, it is possible to generate a scene graph according to a three-dimensional processing type and accumulate all scene graphs to create a tree structure. It is also possible to scan (traverse) all nodes in this tree structure from a root node, and generate three-dimensional scene drawing data to thereby combine and displaying all media. Moreover, a rendering operation can be performed using the three-dimensional canvas means to express the three-dimensional scene drawing data on the Web browser.
Here, it is preferable that the aforesaid three-dimensional scene graph generation means comprises: a means, which is activated by a three-dimensional drawing command from the aforesaid manager means, for setting or changing three-dimensional attributes including material, texture, camera, light and font style in the scene graph, and generating a scene graph with a three-dimensional attribute setting to thereby reflect a change in the three-dimensional attributes in a three-dimensional space.
Also it is preferable that the aforesaid three-dimensional scene graph generation means comprises: means, which is activated by a three-dimensional drawing command from the aforesaid manager means, for setting or changing a behavior of a scene graph, expressing a moving image animation from part of a scene or an entire scene according to a time, an event and/or the like, and generating a scene graph from a three-dimensional behavior setting to thereby reflect a change of the three-dimensional behavior in the three-dimensional space. In this case, the three-dimensional drawing command from the aforesaid manager means comprises: information on a sensor, a timer and an interpolation for defining a behavior of a three-dimensional animation.
According to such a structure, there is provided a means for setting a behavior for a scene graph and expressing the moving image animation from part of a scene or an entire scene according to a time, an event and/or the like. More specifically, there can be provided a means capable of specifying behaviors including walking, sitting, standing, bowing and looking, for a three-dimensional CG character.
According to the one embodiment, the aforesaid three-dimensional scene graph generation means comprises: a two-dimensional embedding means for receiving a three-dimensional drawing command from the aforesaid manager means for embedding two-dimensional image data, and generating a scene graph in which the two-dimensional image data becomes embedded. In this case, it is desirable that this two-dimensional embedding means comprises the means of: receiving from the aforesaid manager means a three-dimensional drawing command which includes an image mapping setting for mapping an image to a three-dimensional image; and generating a scene graph for setting a texture for image mapping in the scene graph tree retained by the aforesaid three-dimensional scene drawing means. Also, it is preferable that the aforesaid two-dimensional embedding means comprises: a hyperlink setting means for receiving hyperlink data as well as two-dimensional image data as a three-dimensional drawing command if a hyperlink is set in a two-dimensional image to be embedded, and generating a scene graph for embedding two-dimensional data which includes hyperlink information therein.
According to such a structure, it is possible to generate an area for displaying two-dimensional rendering in a three-dimensional space, paste a two-dimensional image in the three-dimensional space while retaining two-dimensional attributes of the two-dimensional image, and furthermore specify these expressions within an XML scheme easily.
According to the one embodiment, the aforesaid three-dimensional scene graph generation means comprises: an XVL loader means, which is activated by a three-dimensional drawing command from the aforesaid manger means, for reading an external XVL file, retrieving a lattice-shaped three-dimensional element using a Lattice kernel, converting the lattice-shaped three-dimensional element into a rounded three-dimensional element, and generating a scene graph.
According to such a structure, it is possible to read an XVL file, retrieve a three-dimensional basic element by using a xe2x80x9cLattice kernel meansxe2x80x9d, and generate a scene graph. In this case, there is preferably provided a means for downloading the xe2x80x9cLattice kernel meansxe2x80x9d as a Java class library from a Web server to a client personal computer, and executing the xe2x80x9cLattice kernel meansxe2x80x9d on the Web browser. Moreover, it is desirable that display objects including an XVL-described personage (a three-dimensional CG character) and a studio set can be read, and that a plurality of display objects can be displayed on the same scene graph.
According to the one embodiment, the aforesaid three-dimensional scene graph generation means comprises: a VRML loader means, which is activated by a three-dimensional drawing command from the aforesaid manager means, for reading an external VRML file, analyzing a content of the external VRML file, retrieving a thee-dimensional basic element, and a relationship and a movement of the three-dimensional basic element, and generating a scene graph.
According to such a structure, it is possible to read a VRML file, analyze syntax of the VRML file, retrieve a VRML-described thee-dimensional basic element, and a relationship and a movement of the three-dimensional basic element, and generate a scene graph. Especially, it is desirable that display objects including a VRML-described personage (a three-dimensional CG character) and a studio set can be read, and that a plurality of display objects can be displayed on the same scene graph.
According to the one embodiment, the aforesaid three-dimensional scene drawing means comprises: a three-dimensional scene graph construction means for receiving a plurality of scene graphs according to a content of a three-dimensional drawing command from the aforesaid scene graph generation means, generating a scene graph tree of scenes to be displayed using structured group nodes and reel nodes based on the scene graphs, and concomitantly performing an addition, a deletion or a change for the scene graphs in the aforesaid scene graph tree; and a means for receiving the aforesaid scene graph tree from the aforesaid three-dimensional scene graph construction means, scanning (traversing) all nodes in this scene graph tree from its root node, converting a coordinate system of polygon shape data (a modeling coordinate system), defined in each node, to a three-dimensional coordinate system in a three-dimensional space (a world coordinate system), and thereby generating scene drawing data.
In this case, it is preferable that the aforesaid scene drawing data includes rendering mode data generated from non-polygon shape data retained in the scene graph tree.
Also in this case, it is preferable that there is further included a view control means for receiving a scene graph indicating a content of a camera in the scene graph tree, converting a coordinate to change a viewpoint in a three-dimensional scene, and passing the scene graph with a changed viewpoint to the aforesaid scene drawing means.
According to the one embodiment, the aforesaid three-dimensional canvas means comprises: a view coordinate conversion means for receiving scene drawing data including the polygon shape data from the aforesaid three-dimensional scene drawing means, and normalizing coordinates of the polygon shape data to draw the polygon shape data in two dimensions on the three-dimensional canvas; and a means for receiving normalized projection scene drawing data from the aforesaid view coordinate conversion means, and generating graphic display data while buffering image drawing data to display images eliminated by hidden surface removal in the aforesaid three-dimensional canvas.
In this case, it is desirable that the aforesaid three-dimensional canvas means further comprises: a means for receiving scene drawing data excluding the polygon shape data from the aforesaid three-dimensional scene drawing means, and generating a rendering mode according to the scene drawing data. Also the aforesaid three-dimensional canvas means preferably comprises: a means for generating three-dimensional canvas information based on an operation event from the Web browser for the three-dimensional canvas, and passing the three-dimensional canvas information to the aforesaid scene drawing means.